Automatic color constancy for image sensors

ABSTRACT

An electronic imaging system operates as closely as possible to the cone spectral response space to obtain a human eye-like long, medium, short (LMS) wavelength response. An input image, for example, red-green-blue (RGB), is transformed to an LMS color space similar to the human long-, middle-, and short-wavelength cone receptor responses. Adaptation levels for each LMS component are calculated. The adaptation levels are then used to adjust the sensitivity of each LMS sensor response to obtain an LMS component image. The LMS component image then is transformed back to an RGB component image for further processing or display.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to digital image sensors, and, morespecifically, to a chromatic adaptation method for maintaining colorconstancy under different illuminations in a digital image sensor.

[0003] 2. Brief Description of the Related Art

[0004] Color constancy is one of the characteristics of the human visionsystem. The color appearance of the same object looks approximatelyidentical under vastly different types of natural and artificial lightsources, such as sun light and moon light, and incandescent,fluorescent,, and candle light. The ability of the human visual systemto determine surface colors under this wide range of illuminationconditions is called constancy. In electronic imaging systems, this iscommonly implemented with limited success as an automatic white balance.Extensive research has been conducted into ways to achieve humaneye-like color constancy in electronic image sensors. Nevertheless,present day white balance systems not only lack a response sufficientlysimilar to that of the human eye, they also achieves only a narrowsubset of the overall needs of a true color constancy system.

[0005] One of the ways that the human visual system achieves constancyis referred to as adaptation, which can be understood as a change ingain of the signal from the cone receptors of the eye. The conereceptors become less sensitive because of chemical bleaching in thepresence of increased light. This results in a reduction in sensitivityfor those cones experiencing greater light intensities. If the light isstrongly colored, then the different cone types will becomedifferentially adapted. In red light, for example, long wavelength coneswill become less sensitive. The effect of adaptation is to make the eyehave a sensitivity range appropriate to the environment.

[0006] This theory of constancy in the human vision system generallyholds that differences in the type of illumination are accommodated bythe chromatic adaptation of the human vision system. The sensitivitiesof long (L), middle (M) and short (S) wavelength cones adapt to stimuliin a largely independent way. This is the hypothesis proposed by vonKries, although exact details of the adaptation were not provided.(“Chromatic Adaptation,” J. von Kries, Festschrift derAlbrecht-Ludwig-Universität, 1902).

[0007] Examples of some of the algorithms that have been explored forproviding color constancy in electronic image sensors include: GrayWorld, Retinex, Gamut Mapping Methods, Color by Correlation, and NeuralNet Methods. See “A Comparison of Computational Color ConstancyAlgorithms,” Parts One and Two, by K. Barnard et al., available athttp://www.cs.berkeley.edu/˜kobus/research/publications/comparison{_(—)1or _(—)2}/comparison{_(—)1 or _(—)2}.pdf.

[0008] Most electronic image sensors are designed with spectralresponses that evenly divide the visible spectrum into color ranges,such as the three primary colors red, blue, and green, with little or nooverlap between each range. The response represents the absolute photonacquisition experienced by each pixel of the digital image sensor, forexample.

[0009] In contrast to the known electronic image sensors, the threetypes of color receptors in the human eye—long-, middle-, andshort-wavelength cones (LMS)—have been found to exhibit significantoverlap in spectral response. As a consequence of this overlappingspectral response, the hue-discrimination response of the human eye ishighly non-linear, with peak sensitivity occuring near certainwavelengths. By comparison, a imaging array that utilizes an RGB filter,such as a Bayer filter, acts as a simplified band pass filter that doesnot correspond to the spectral response of the human eye.

[0010] Color standards are maintained by the Commission Internationalede L'Eclairage (CIE). The CIE has developed standard color systems basedon the concept of a standard observer. The standard observer is based ona model of human rods and cones. The CIE system does not take adaptationinto account, however. The CIE systems define color using tristimulusvalues X, Y, and Z. Y is the same as luminance (black and white).

[0011] It would be desirable to have an imaging system which more nearlyreplicates the color discrimination of the human eye to achieve moreconstancy in color reproduction under different lighting conditions.

BRIEF SUMMARY OF THE INVENTION

[0012] The present invention provides a method, apparatus, and storedprogram for an electronic imaging system to operate as close as possibleto the cone spectral response space of the human eye. Accordingly, thepresent invention provides a chromatic adaptation method and system bywhich color constancy under different illuminations can be more closelymaintained.

[0013] According to an exemplary embodiment of the present invention, aninput image having an imager color space, for example, a red-green-blue(RGB) color space, is transformed to an LMS color space, a color spacesimilar to the human long-, middle-, and short-wavelength cone receptorresponses. Mean and maximum adaptation levels (signal strengths) foreach LMS component are calculated. The adaptation levels are then usedto equalize the sensitivity of each LMS sensor response to obtain anequalized LMS component image. This is equivalent to an automatic gaincontrol for each individual LMS sensor output, and compensates fordifferent types of illumination. The LMS component image is thenconverted back to a display device color space, e.g., an RGB image. Theinvention may be embodied as an algorithm executed in hardware,software, or a combination of the two.

[0014] The invention achieves significant color constancy behavior, andis straightforward and relatively easy to implement.

[0015] These and other features and advantages of the invention will bebetter understood from the following detailed description, which isprovided in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016]FIG. 1 is a flow chart illustrating a process for implementingcolor constancy system according to an exemplary embodiment of thepresent invention.

[0017]FIG. 2 is a flow chart illustrating in greater detail portions ofthe process illustrated in FIG. 1.

[0018]FIG. 3 illustrates a CMOS imaging system including circuitry forcomputing color constancy according to an exemplary embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

[0019] In the following description, an exemplary imager color spacewill be described as an RGB color space; however, the invention hasapplicability to other input device color spaces as well.

[0020] Referring to FIG. 1, image sensor data from an RGB color space istransformed to an LMS color space by the sequential steps of:

[0021] Setting up an RGB to LMS color space transformation matrix asshown in step 2;

[0022] transforming an inputted RGB component image to an LMS componentimage using the transformation matrix in steps 3 and 4;

[0023] computing an adaptation level (mean signal strength) andequalizing the sensitivity for each LMS component image in step 6;

[0024] computing a composite gain factor based on a maximum adaptationlevel (signal strength) for each LMS component image in step 8;

[0025] applying the composite gain factor for each LMS component imagein step 10;

[0026] transforming the adjusted LMS component image back to an RGBcomponent image in step 12; and

[0027] displaying or otherwise outputting the RGB image in step 14.

[0028] The process sequence illustrated in FIG. 1 is described below infurther detail in relation to an exemplary embodiment of the presentinvention. Referring first to step 2, based on the chromaticity of atargeted display monitor, a 3 by 3 transformation matrix can be derivedto transform the image sensor data from RGB space to CIE tri-stimulusXYZ space. See “Fundamentals of Three Dimensional Computer Graphics” byAlan Watt, 1989 AddisonWesley, ISBN 0-201-154420, the disclosure ofwhich is incorporated herein by reference. Because the CIE XYZcolor-matching functions all represent a linear combination of cone(LMS) responses, the transformation from an XYZ color space to an LMScolor space can also be defined by a 3 by 3 linear matrix. See “ColorAppearance Models” by Mark D. Fairchild, 1997 Addison-Wesley, ISBN0-201-63464-3, the disclosure of which is incorporated herein byreference. A composite RGB to LMS color space transformation matrix thuscan be formed by concatenating the two 3 by 3 matrices. The resultingconcatenated matrix is obtained by multiplication of the two 3 by 3matrices.

[0029] The color space conversion process is applied to every normalizedinput RGB pixel to obtain the pixel values under LMS color space. Forexample, using an eight bit system, RGB intensity values will range from0-255. These values are transformed into the LMS color space to have anintensity value from 0 (black) to 1 (saturated), for example, whereby anormalized value of input RGB pixels is used to divide each input RGBpixel value by the maximum permitted value.

[0030] Equation (1) is an exemplary transformation matrix for an RGBimaging device having an ITU-R709 (SMPTE 274M) chromaticity:$\begin{matrix}{\begin{bmatrix}L \\M \\S\end{bmatrix} = {\begin{bmatrix}0.134 & 0.640 & 0.047 \\0.155 & 0.758 & 0.087 \\{- 0.019} & 0.153 & 0.877\end{bmatrix} \times \begin{bmatrix}R \\G \\B\end{bmatrix}}} & (1)\end{matrix}$

[0031] This transformation matrix is established in step 2 of FIG. 1,and is used in step 4 to transform input RGB pixel data obtained at step3 in FIG. 1.

[0032] Once the input RGB image is converted to an LMS image in step 4,an adaptation model is determined for each of the L, M, and S componentsof the LMS color space. A central issue of any adaptation model is tofind the appropriate adaptation level for each LMS component. Accordingto an exemplary embodiment of the present invention, use is made of theroot of a relatively high degree geometric mean. It provides a veryrobust measure of the adaptation level.

[0033] The high degree geometric mean for each L, M, and S component isobtained by initially calculating a geometric sum (gsum), as shown inequation (2), by summing each pixel intensity value pi raised to thepower of K. K preferably is at least about 4, more preferably 5 or 6.$\begin{matrix}{{gsum} = {\sum\limits_{i = 1}^{N}{{pow}( {{P\quad i},K} )}}} & (2)\end{matrix}$

[0034] Once the geometric sum for each component L, M, and S iscalculated, the root mean for each component is determined as shown inequation (3), by averaging the result (gsum) of equation (2), and takingthe Kth root of that average, as follows: $\begin{matrix}{{{adapt} - {level}} = {{pow}\quad ( {{{gsum}/N},{1.0/K}} )}} & (3)\end{matrix}$

[0035] In equations (2) and (3), Pi is the value of each LMS pixel i,where i varies from 1 to N, N being the total number of pixels for eachLMS image component, and K is a power factor. Having K around at least 4works quite well across wide spectrum of test images. The value of“adapt-level” is calculated independently for each LMS component(L_adapt-level, M_adapt-level, S_adapt-level).

[0036] After the adaptation model values L_adapt-level, M_adapt-level,S_adapt-level are calculated in step 6, they are used to compute acomposite gain factor made up of a global gain factor, and individualcomponent gain factors for each LMS image component. The composite gainfactor is applied to each of the LMS components in step 10. The globaland individual component gain factors are combined into the compositegain factor for efficient image processing pipeline implementation.

[0037] The global gain factor is related to the exposure time. It can beviewed as a form of automatic exposure control in a digital camera, andis common to all LMS components. The individual component gain factor isrelated to the chromatic adaptation. The two gain factors areindependent from each other.

[0038] The composite gain factors are obtained by determining thecorrect gain balance between each of the L, M, and S components, basedon maximum gain values, and determining the overall, global gain. Morespecifically, the composite gain factor for each LMS component iscalculated as shown in FIG. 2 and described as follows:

[0039] Equalization factors “L-equalize”, “M-equalize”, and “S-equalize”are obtained in the following steps:

[0040] Using the results of the adaptation model (Equation 3 above), themaximum adapt-level value is determined from the three adapt-levelvalues for the LMS components, as shown in step 8 a of FIG. 2. Thisadapt value is denoted adapt_max. For the component L, M, or S with thismaximum “adapt-level” value, the equalization factor equals one (step 8b).

[0041] For the two LMS components other than the one with maximum“adapt-level” value (adapt_max) the respective equalization factorL-equalize, M-equalize or S-equalize is related to adapt_max asadapt_max/(L, M, or S)-adapt_level. See step 8 c of FIG. 2. Theequalization factors will be used to balance the sensitivity of thethree components. Consequently, pixels of the component having thehighest intensity will remain at the same value, while pixels of theother components will have their intensities increased. For example,when an incandescent light source is being used, the light will havehigher intensities in the L wavelengths, and the equalization factor forthe L wavelengths will be one. Since intensity values range from 0 to 1,the other wavelength components—M and S—will have equalization factorsgreater than one. Accordingly, their sensitivities will be increasedrelative to the L wavelengths, thereby simulating the manner in whichthe human eye provides constancy.

[0042] A global gain control is obtained by first determining a maximumpixel value for each LMS image component in step 8 d. These values aredenoted as L_max, M_max, and S_max.

[0043] For each of the LMS components, the equalization factor isapplied to the maximum pixel value to obtain equalized maxima, i.e.,L_max_equalized equals L_max multiplied by the equalization factor“L_equalize” found in the previous steps, as shown in step 8 e of FIG.2.

[0044] The results of step 8 e, are compared to obtain the maximum valueof L_max_equalized, M_max_equalized, and S_max_equalized, which isdenoted as LMS_max in step 8 f. The global gain control is found in step8 g as global_gain=1.0/LMS_max.

[0045] The composite gain factor for each LMS component is calculated instep 8 h by multiplying the global_gain to the adaptation factor of eachcomponent. For example, the composite gain for the L component is:L_gain=global_gain×L_adapt.

[0046] The global gain factor is combined with the component gain factorto provide a composite gain factor which is applied at step 10 of FIG. 1to each LMS component image. For example, for each pixel in the Lcomponent image, the final pixel value is obtained by multiplying with“L_gain”. Similarly, the final pixel value for each pixel in the Mcomponent image is multiplied by “M_gain” and the S component “S_gain.”

[0047] The LMS image is then transformed to the RGB color space andscaled by the desired bit precision, into an 8-bit domain, for example,for integer representation by an inverse application of equation (1)above using matrix multiplication.

[0048] The invention provides an electronic imaging system that operatesas closely as possible to the cone spectral response space to obtain ahuman eye-like response. The inventive algorithm is embodied in hardwareor software and executes a chromatic adaptation by which color constancyunder different illuminations is maintained. An input red-green-blue(RGB) image, for example, is transformed to an LMS color space similarto the human long-, middle-, and short-wavelength cone receptorresponses. Adaptation levels for each LMS component are calculated,which are then used to adjust the sensitivity of each LMS sensorresponse to obtain an LMS component image. The LMS component image thenis transformed back to an RGB component image for further processing ordisplay.

[0049] Although the invention has been described mostly using imagesacquired in an additive, RGB color space, other color imaging protocolscould be used in the present invention, including, for example, asubtractive CMY color space. Similarly, although the example of atransform matrix from RGB space to CIE tri-stimulus XYZ space wasutilized, with a second linear transform from XYZ to LMS defined by a 3by 3 linear matrix, the invention need not be limited to the particularcalorimetric color spaces or types of transforms.

[0050]FIG. 3 illustrates a processor system 20 in which an imagingdevice 22 incorporating the color constancy methods according to thepresent invention is utilized. System 20 may be a stand alone system, ora system of individual, interconnectable components, each of thecomponents incorporating one or more of the various subsystemsillustrated in FIG. 3.

[0051] System 20 includes a CPU 24 and a user input/output (I/O) device26 connected to a system bus 28. System 20 also includes MRAM 30. MRAM30 communicates with the other components by way of system bus 28. Otheroptional peripheral devices include a disk drive 32 and a CD ROM drive34. Additional optional peripheral devices could include removablememory storage devices for storing images obtained by the CMOS imager,such as memory cards, memory sticks, etc.

[0052] While preferred embodiments of the invention have been describedand illustrated above, it should be understood that these are exemplaryof the invention and are not to be considered as limiting. For example,although an exemplary embodiment has been described in connection with aCMOS image sensor, the invention is appicable to other electronic imagesensors, such as CCD image sensors, for example. Additions, deletions,substitutions, and other modifications can be made without departingfrom the spirit or scope of the present invention. Accordingly, theinvention is not to be considered as limited by the foregoingdescription but is only limited by the scope of the appended claims.

What is claimed as new and desired to be protected by Letters Patent ofthe United States is:
 1. A method for improving color constancy underdifferent illumination conditions, comprising: transforming an inputimage to an image in a color space corresponding to that of a human eye,the color space image having a plurality of components; calculatingadaptation levels for each of the plurality of components; adjusting asensitivity of each of the plurality of components based on theadaptation levels; and transforming the color space image to an outputimage.
 2. A method as in claim 1, wherein the input image originatesfrom an image sensor.
 3. A method as in claim 1, wherein the input imageoriginates from a digital image sensor.
 4. A method as in claim 1,wherein the input image originates from a CMOS image sensor.
 5. A methodas in claim 1, wherein the input image is an RGB image.
 6. A method asin claim 5, wherein the color space corresponding to that of a human eyeis an LMS color space.
 7. A method as in claim 5, wherein the inputimage is an RGB image which is tranformed to an LMS color space.
 8. Amethod as in claim 5 wherein the input image is an RGB image which istransformed using a transformation matrix representing the concatenationof an RGB to CIE tri-stimulus XYZ space transform and a CIE tri-stimulusXYZ space to LMS transform.
 9. A method as in claim 5, wherein theadaptation levels are calculated for each LMS component.
 10. A method asin claim 1, wherein the adaptation levels are calculated as root meansof individual pixel intensity values.
 11. A method as in claim 10,wherein the root means are calculated by obtaining a geometric sum ofthe pixel intensity values to the Kth power, and finding the Kth root ofthe geometric sum, as follows: $\begin{matrix}{{gsum} = {\sum\limits_{i = 1}^{N}{{pow}( {{P\quad i},K} )}}} \\{{{adapt} - {level}} = {{pow}\quad ( {{{gsum}/N},{1.0/K}} )}}\end{matrix}$

where Pi denotes a pixel intensity value for each of a plurality ofpixels i, i ranging from 1 to N, with N being a total number of pixels,and K is a high degree power factor.
 12. A method as in claim 11,wherein K is at least about
 4. 13. A method as in claim 1, furthercomprising using the adaptation levels to calculate a composite gainfactor for each LMS component, and using the composite gain to adjustthe sensitivity of each of the plurality of components.
 14. A method asin claim 11, wherein calculating the composite gain factors includesdetermining a maximum pixel value for each of the plurality ofcomponents to obtain a global gain value portion at the composite gainfactors.
 15. A method as in claim 10, wherein calculating the compositegain factors further includes assigning an adaptation factor of one tothe component with the highest maximum value.
 16. An imager comprising:means for transforming an input image to an image in a color spacecorresponding to that of a human eye, the color space image having aplurality of components; means for calculating adaptation levels foreach component; means for adjusting the sensitivity of each componentbased on the adaptation levels; means for transforming the adjustedcolor space image to an output image.
 17. An imager as in claim 15,wherein the means for transforming, calculating, and adjusting comprisesone of hardware, software, or a combination of hardware and software.18. An imaging device comprising: an imager providing an input image; atransforming unit for transforming an input image from the imager to animage in a color space corresponding to that of a human eye, the colorspace having a plurality of components; a calculating unit forcalculating adaptation levels for each component; an adjusting unit foradjusting the sensitivity of each component based on the adaptationlevels; and a transforming unit for transforming the adjusted colorspace image to an output image.
 19. An imaging device as in claim 18,further comprising a display device having a screen, and circuitry fordisplaying the RGB component image on the screen.
 20. An output imagecomprising: an image produced by the method of transforming an inputimage to an image in a color space corresponding to that of a human eye,the color space having a plurality of components; calculating adaptationlevels for each of the plurality of components; adjusting a sensitivityof each of the plurality of components based on the adaptation levels;and transforming the adjusted color space image to the output image. 21.A storage medium containing program for use in maintaining colorconstancy under different illumination conditions in an imager sensor,the program comprising instructions for transforming an input image toan image in a color space corresponding to that of a human eye, thecolor space having a plurality of components; calculating adaptationlevels for each of the plurality of components; adjusting a sensitivityof each of the plurality of components based on the adaptation levels;and transforming the adjusted color space image to an output image. 22.A method for improving color constancy under different illuminationconditions, comprising: computing a color space transformation matrix;transforming an input image to a transformed image having pluralcomponents using the color space transformation matrix; computingadaptation levels for the transformed image; computing a composite gainfactor based on the adaptation levels; applying the composite gainfactor to the transformed image to obtain an modified transformed image;transforming the modified transformed image to an output image; andprocessing the output image.
 23. A method as recited in claim 22,wherein the color space input image is an RGB color space image, and thetransformed image has a color space similar to that of the human eye.24. A method as recited in claim 22, wherein the composite gain factorincludes a global gain factor and an individual component gain factorfor each of the plurality of components.
 25. A method as recited inclaim 22, where the output image is an RGB image, and the step oftransforming the transformed image to an output image comprisesmultiplying the modified transformed image using an inverse of thetransformation matrix.
 26. A method as recited in claim 22, wherein thestep of applying adaptation levels to compute a composite gain factorcomprises: determining a maximum of the adaptation levels; obtainingadaptation factors for each of the plurality of components; obtainingmaximum pixel intensity values for each of the plurality of components;multiplying the maximum pixel values by the respective adaptation valuesand selecting a maximum result; computing a global gain factor based onthe maximum result; and calculating the composite gain factor as aproduct of the global gain factor and the respective adaptation factor.27. An imaging device having improved color constancy under differentillumination conditions, comprising computing elements for obtaining acolor space transformation matrix, transforming an input image to atransformed image having plural components using the color spacetransformation matrix, computing adaptation levels for the transformedimage, computing a composite gain factor based on the adaptation levels,applying the composite gain factor to the transformed image to obtain anmodified transformed image, transforming the modified transformed imageto an output image; and processing elements for processing the outputimage.
 28. An imagaing device as in claim 27, wherein the color spaceinput image is an RGB color space image, and the transformed image has acolor space similar to that of the human eye.
 29. An imaging device asin claim 27, wherein the composite gain factor includes a global gainfactor and an individual component gain factor for each of the pluralityof components.
 30. An imaging device as in claim 27, where the outputimage is an RGB image, and transforming the transformed image to anoutput image comprises multiplying the modified transformed image usingan inverse of the transformation matrix.
 31. An imaging device as inclaim 27, wherein applying adaptation levels to compute a composite gainfactor comprises: determining a maximum of the adaptation levels;obtaining adaptation factors for each of the plurality of components;obtaining maximum pixel intensity values for each of the plurality ofcomponents; multiplying the maximum pixel values by the respectiveadaptation values and selecting a maximum result; computing a globalgain factor based on the maximum result; and calculating the compositegain factor as a product of the global gain factor and the respectiveadaptation factor.
 32. An imaging device as in claim 27, wherein theimage device comprises a CMOS imager.
 33. A storage medium containing aprogram for improving color constancy under different illuminationconditions, the program comprising instructions for computing a colorspace transformation matrix; transforming an input image to atransformed image having plural components using the color spacetransformation matrix; computing adaptation levels for the transformedimage; computing a composite gain factor based on the adaptation levels;applying the composite gain factor to the transformed image to obtain anmodified transformed image; and transforming the modified transformedimage to an output image.
 34. A storage medium as in claim 33, whereinthe color space input image is an RGB color space image, and thetransformed image has a color space similar to that of the human eye.35. A storage medium as in claim 33, wherein the composite gain factorincludes a global gain factor and an individual component gain factorfor each of the plurality of components.
 36. A storage medium as inclaim 33, where the output image is an RGB image, and the step oftransforming the transformed image to an output image comprisesmultiplying the modified transformed image using an inverse of thetransformation matrix.
 37. A storage medium as in claim 33, wherein thestep of applying adaptation levels to compute a composite gain factorcomprises: determining a maximum of the adaptation levels; obtainingadaptation factors for each of the plurality of components; obtainingmaximum pixel intensity values for each of the plurality of components;multiplying the maximum pixel values by the respective adaptation valuesand selecting a maximum result; computing a global gain factor based onthe maximum result; and calculating the composite gain factor as aproduct of the global gain factor and the respective adaptation factor.